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The lack of attention given to research accomplishments is troubling because 
the resulting assessments and rankings of institutions will overlook an 
important facet of their mission. This study shows how readily available data 
from the Institute of Scientific Inquiry (ISI) may be used to estimate the 
number of scholarly articles written by an institution's faculty for 
virtually any year from 1990 forward. Data from the ISI databases were used 
to estimate the total number of publications attributed to individuals at 
each of approximately 1,300 4-year colleges and universities in the United 
States for 1996. The study also constructed a standardized measure of 
research output by dividing total publications by the number of full-time 
faculty at the institution. The study shows how institutions are ranked 
according to these two publication measures, how the measures vary by type of 
institution, and how they are correlated with other selected measures of 
research resources and institutional quality. (Contains 3 figures, 5 tables, 
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Using Publications Counts to Measure an Institution’s Research Productivity 



Abstract 



Studies that address institutional-level productivity and publications that rank institutions 
often base their analysis solely on teaching activities, and overlook the research activities 
of an institution. The lack of attention given to research accomplishments is troubling 
because the resulting assessments and rankings of institutions will overlook an important 
facet of their mission. In this paper, we show how readily available data from the 
Institute of Scientific Inquiry (ISI) may be used to estimate the number of scholarly 
articles written by an institution’s faculty for virtually any year from 1990 forward. We 
also construct a standardized measure of research output by dividing total publications by 
the number of full-time faculty at the institution. We will show how institutions are 
ranked according to these two publication measures, how the measures vary by type of 
institution, and how they are correlated with other selected measures of research 
resources and institutional quality. 
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Using Publications Counts to Measure an Institution’s Research Productivity 



Introduction 

One of the unique features of institutions of higher education (IHEs) is the multi- 
product nature of the organization. It is widely acknowledged that the role of faculty and 
their institutions is to engage in the production of teaching, research, and service (Golden 
and Carstensen, 1992). While education stakeholders such as legislators, parents, and 
(undergraduate) students focus primarily on the value of teaching activities of IHEs, 
faculty in doctoral-granting universities tend to be as concerned, if not more concerned, 
with their research activities and accomplishments. Full-time faculty on average 
published nearly two refereed or juried publications per year for 1997 and 1998, with the 
bulk of this activity being concentrated in research- and doctoral-level institutions (U.S. 
Department of Education, 2001b, Table 28). Institutions devote considerable resources to 
the production of research, spending a reported $19.5 billion on research in 1996-97 
(U.S. Department of Education, 2001a, Tables 340 and 341). This figure substantially 
understates the financial commitment to research activities since faculty salaries are 
usually classified as an “instructional” expense and faculty report that over 25 percent of 
their work time is devoted to research (U.S. Department of Education, 2001b, Table 2). 
The federal government appropriated close to $16 billion in 1997 to support university 
research (U.S. Department of Education, 1997). 

There have been numerous studies performed over time to examine various 
aspects of the productivity of faculty members, academic departments, and entire 
institutions in higher education. By far, the majority of individual- and departmental- 
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level studies have focused on measuring the research productivity of faculty. 

Assessments of individual and departmental research accomplishments are most often 
based at least in part on the number of publications produced over a specific time period 
(Tan, 1986). In his review of the literature on assessment efforts in higher education, Tan 
(1986) was critical of these studies for focusing almost exclusively on faculty research 
productivity and neglecting other aspects of quality. 

Interestingly, studies that address institutional-level productivity often base their 
analysis solely on teaching activities, and thus overlook the research activities of an 
institution. This applies to both scholarly research studies as well as commercial 
publications such as US News and World Report ( USNWR ) and Money Magazine that 
rate institutions on the basis of readily-accessible data on variables such as their 
expenditures per student and graduation rates. This also extends to institutions that use 
“performance indicators” to evaluate the effectiveness of their operations. Toutkoushian 
and Danielson (2002) note that “The goal of advancement of knowledge through research 
is especially underrepresented among the sets of indicators that are used in higher 
education.” Even institutional-level studies that use research measures as inputs, such as 
higher education cost studies, tend to rely on proxy measures for research productivity, 
such as dollars of sponsored research funding, rather than the same types of research 
output measures used in individual- and departmental-level studies (Nedwek, 1996; 

Cohn, Rhine, and Santos, 1989; de Groot, McMahon, and Volkwein, 1991). 

Furthermore, since these measures are rarely expressed on a per-capita basis, they tend to 
be biased in favor of larger institutions. 
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The focus on teaching-related measures and absence of research-related measures 
in institutional-level studies is driven by several factors. First, many of the higher 
education stakeholders who advocate for more accountability in higher education are 
primarily interested in the teaching functions of IHEs. This would certainly include 
college-bound high school students and their families who are looking for information to 
guide them in their college selection process. Secondly, information on the research 
output of institutions has not been readily available to those interested in evaluating and 
comparing institutions. Performance indicator systems, college rankings, and empirical 
studies are often forced to rely on accessible metrics such as expenditures per student, 
graduation rate, student-faculty ratios, and dollars received for sponsored research. 

The lack of attention given to research accomplishments is troubling because the 
resulting assessments and rankings of institutions will overlook an important facet of 
their mission. Furthermore, if teaching and research tend to be substitute rather than 
complementary activities (Fox, 1992), then schools that give relatively more emphasis to 
research may appear to be less productive than other institutions because this aspect of 
productivity is not being measured. In the long run, such a situation could give rise to 
increased pressure from parents and state governments to give greater emphasis to 
teaching outcomes, and as a result the production of research may suffer (see for example 
the empirical results in Bellas and Toutkoushian (1999), Fox (1992), and Porter and 
Umbach (2001)). This is especially problematic in many of the humanities and social 
science disciplines where faculty members cannot rely on federal research dollars to 
support their research activities. 
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In this paper, we show how readily available data from the Institute of Scientific 
Inquiry (ISI) may be used to estimate the number of scholarly articles written by an 
institution’s faculty. While there are some caveats and qualifiers with our approach, the 
resulting counts give rise to a reasonable proxy measure of an institution’s research 
output that can be calculated for virtually any IHE and any year from 1990 forward. We 
also construct a standardized measure of research output by dividing total publications by 
the number of full-time faculty at the institution. We will show how institutions are 
ranked according to these two publication measures, how the measures vary by type of 
institution, and how they are correlated with other selected measures of research 
resources and institutional quality. 

Literature Review 

The literature on faculty research productivity is quite voluminous and we will not 
attempt a comprehensive review of all relevant studies here. The vast majority of studies 
in this group have used either individual-level or departmental-level data. These studies 
would include those that have tried to explain variations in research productivity, as well 
as others that have used research productivity measures as independent variables in 
models of faculty earnings (e.g., Ransom and Megdal, 1993) and departmental 
reputations (e.g., Ehrenberg and Hurst, 1996). For our purpose, we will focus on how 
individual- and departmental-level studies have represented research productivity. We 
then turn to studies using institutional-level data, and review how they have represented 
research productivity in their analysis. Finally, we review other institutional-level 
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publications and studies, such as the USNWR rankings, performance indicators and cost 
studies. 

Individual- and Departmental-Level Studies. 

By far, the most commonly-used measure of individual and departmental research 
productivity is the number of faculty publications in selected outlets such as academic 
journals (Baird, 1991; de Groot, McMahon, and Volkwein, 1991; Dundar and Lewis, 
1998; Golden and Carstensen, 1992; Johnes and Johnes, 1995; Jordan, Meador, and 
Walters, 1989; Toutkoushian, Dundar, and Becker, 1998; Baird, 1986; Creamer, 1998; 
Fox, 1992; Porter and Umbach, 2001; Tien and Blackburn, 1996; Wanner et al., 1981), or 
a summative index constructed from counts of conference papers, journal publications 
and books (Buchmueller et al., 1999; Noser, et al., 1996; Perry et al., 2000; Bellas and 
Toutkoushian, 1999). Usually these are limited to a specific period of time, and are not 
adjusted for prestige of publication source or multiple authorships. See Tan (1986) for a 
more thorough review of the literature. It has also become quite common for researchers 
to rank departments within a specific field on the basis of their total publications (Moore, 
1973; Miller and Tollison, 1975; Bell and Seater, 1978; House and Yeager, 1978; Graves, 
Marchand, and Thompson, 1982; Laband, 1985; Borokhovich, Bricker, Brunarski, and 
Simkins, 1995). 

Studies focusing on publication counts have, however, been criticized because 
they vary across disciplines due to the nature of the work being performed and the 
conventions for communicating research (Wanner, Lewis, and Gregorio, 1981), and do 
not take into account the quality of research (Braxton and Bayer, 1986), except to the 
extent that it has passed through peer review (Manis, 195 1). In response, some scholars 
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have used the number of citations received by an individual faculty member to measure 
their research productivity (e.g., Long and McGinnis, 1981; Lindsey, 1989; 

Toutkoushian, 1994; Laband, 1985; Diamond, 1986) as well as rank departments (Gerrity 
and McKenzie, 1978; Davis and Papanek, 1984). Citation counts are also not without 
their concerns and limitations (see Toutkoushian, 1994). Other studies, such as Baird 
(1986), Gander (1999), Porter and Umbach (2001), have used the level of research grant 
dollars received as a measure of research activities. 

Institution-level Studies and Rankings. 

In contrast to the literature cited above, fewer studies compare institutions on the 
basis of institutional-level research data. The practice of using empirical data to 
compare and rank institutions dates back to Hughes (1925, 1934), who surveyed scholars 
in specific fields and used the results to rate 38 institutions on the basis of their aggregate 
ratings. Similar procedures were used by Keniston (1959), Webster (1983), Patrick and 
Stanley (1998), and Keith (1999). These studies all relied on subjective (survey) data to 
rate institutions. 

One of the first efforts to use objective data for ranking institutions is by Albert 
Bowker (1965). Bowker rated institutions on the basis of several criteria, including the 
number of faculty who have received the American Council of Learned Societies Award, 
or been named a Woodrow Wilson fellow, and the number of students attending the 
institution who have won National Science Foundation awards. Brown (1967) also 
devised college ratings on the basis of their average ranking for eight measures: (1) 
percentage of faculty possessing a doctorate degree, (2) average faculty compensation, 

(3) percentage of students going on to graduate school, (4) percentage of students at the 
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graduate level, (5) total library volumes per student, (6) number of full-time faculty, (7) 
student-faculty ratio, and (8) current income per student. Adams and Krislov (1978) 
argued in favor of ranking institutions on the basis of average faculty salaries. They 
claimed that this approach had the advantage of readily-accessible data, and that faculty 
salaries were a good measure of faculty - and hence institutional - quality. 

More recently, TheCenter at the University of Florida produced a report titled The 
Top American Research Universities (Lombardi, Craig, Capaldi, Gater, & Mendonca, 
2001). In this report, the authors derived rankings for universities based on nine 
measures that they argued were indicators of quality for research universities: (1) 
research expenditures, (2) federal research funding, (3) endowment assets, (4) alumni 
donations, (5) number of National Academy of Sciences members, (6) number of faculty 
receiving other awards, (7) number of doctorate degrees granted, (8) number of 
postdoctoral appointees, and (9) median SAT score of freshmen. TheCenter ranked 
institutions on the basis of the number of measures in which the institution was rated in 
the top 25 nationally. Interestingly, TheCenter does not utilize any direct measures of 
research output such as publication counts in their rankings of research universities. In 
contrast, Zheng and Stewart (2002) use data on faculty publications, citations, and 
research dollars awarded to rank public research universities using data envelopment 
analysis. Their analysis relies on a causal model to predict research output, and 
institutions are then ranked according to their efficiency in producing output. 

Finally, another set of research studies rely on institutional-level data to explain 
variations in costs per unit of output in higher education (e.g., Brinkman, 1981; Cohn, 
Rhine, and Santos, 1989; de Groot, McMahon, and Volkwein, 1991; Koshal and Koshal, 
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1999). These studies are relevant to the current discussion since they recognized early on 
that IHEs produce outputs in the areas of teaching and research, and thus had to address 
the measurement of institution-level research output. The most commonly adopted 
approach was to use sponsored research dollars or contracts as a measure of research 
activities/productivity (Southwick, 1969; Brinkman, 1981; Cohn, Rhine, and Santos, 
1989; Toutkoushian, 1999; Koshal and Koshal, 1999). 

A notable exception is de Groot, McMahon, and Volkwein (1991), who used 
publication counts for graduate programs in 147 doctorate-granting institutions. The 
data, made available through a study conducted by the Conference Board of Associated 
Research Councils (Jones et al., 1982), accounted for approximately 58% of all 
publications according to the authors of the study (de Groot et al., 1991). Webster and 
Skinner (1996) and Diamond and Graham (2000) have also highlighted rankings of 
institutions on the basis of the graduate program assessments from the 1995 NRC study. 
One major limitation of using the graduate program survey as a means for collecting 
publication data is that there are substantial intervals between surveys (the two NRC 
studies were conducted in 1982 and 1993), and thus researchers may not be able to use 
current data in their analyses. 

Two quick observations are worth noting at this point. First, in contrast to 
departmental-level studies, studies that attempt to evaluate and rank institutions often 
overlook the research activities and accomplishments of the faculty. When these studies 
do acknowledge research, it is usually through metrics that can be readily obtained, such 
as total spending on research. These measures, however, often represent the resources 
available for producing research rather than the quantity or quality of research actually 
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produced by the institution. Second, the wide variation in measures used to examine 
institutional quality suggests that developers of measures are not relying on a theoretical 
framework for the selection of measures to evaluate institutions. Such a framework 
would suggest that the measures be related to the goals and objectives of the 
organization, and accordingly research measures should be included along with teaching 
measures in the analysis. 

The practice of evaluating and comparing EHEs continues today in the form of 
ratings by commercial entities such as USNWR. The USNWR ratings are based on a 
weighted-average score derived from the following measures: (1) academic reputation, . 
(2) retention and graduation rates, (3) faculty salaries, (5) student-faculty ratio, (6) 
percent full-time faculty, (7) percent small and large classes, (8) high school rank, SAT 
scores, acceptance rate and yield rates for freshmen, (9) expenditures per student, and 
(10) alumni giving rate. As noted by Graham and Thompson (2001), the USNWR 
rankings tend to measure the “school’s wealth, reputation, and the achievement of the 
high school students it admits.” (p.3). Even vocal critics of the USNWR rankings admit, 
however, that prospective students often use this information to help them decide where 
to attend college. Empirical work has verified the strong impact that USNWR rankings 
have on admissions outcomes such as proportion admitted and proportion matriculating 
(Monks and Ehrenberg, 1999). 

There are also many instances where analysts rely on institutional-level variables, 
commonly referred to as “performance indicators,” in an attempt to evaluate and compare 
institutions. The general practice in higher education applications is to first identify a set 
of specific indicators, and then compare the values of these indicators to other institutions 
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and track changes in them over time (see, for example, Banta and Borden, 1994; Gaither, 
Nedwek, and Neal, 1994). The focus on empirical data means that the analyst must be 
able to calculate the indicator for his/her institution and for other institutions to which 
comparisons will be made. 

The interest in assessment has seemingly taken on a life of its own in the past ten 
years. Most IHEs now rely on indicators to monitor how well they are doing, and states 
such South Carolina tie the level of higher education funding to such indicators (South 
Carolina Commission on Higher Education, 2002). Proponents of such assessment 
efforts argue that by tracking designated indicators, institutions can . .take steps to 
improve their competitive position” (Taylor and Massey, 1996, p.xv), and determine 
“...whether the college or university is accomplishing its goals” (Dolence and Norris, 
1994, p.64). With this in mind, it is curious to observe that most of the performance 
indicators in use in higher education have very little, if any, direct connection to the 
research mission of postsecondary institutions. For example, of the over 100 indicators 
identified by Taylor and Massy (1996), only one variable — federal grants and contracts 
as a percentage of total current funds revenues - even remotely serves as a proxy for the 
research productivity of faculty. Likewise, most of the examples of performance 
indicator systems described by Banta and Borden (1994) and Gaither, Nedwek, and Neal 
(1994) do not include the publishing frequency of faculty. 

Why, then, do institutional-level research studies, institutional rankings, and 
performance indicator systems rarely if ever utilize information on research publications 
when the literature on individual and departmental productivity clearly suggests that they 
should matter? In large part, the problem is that data on institution-level research 
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productivity has been difficult to obtain. As noted by Cohn et al. (1989), “Ideally, one 
should measure the various components of research output, such as publications, research 
reports, patents, public lectures, and other results of one’s research work. Such a measure 
of output would be hard enough to produce for a single academic department, let alone 
for a large sample of IHEs.” (p.285). Studies of individual- or departmental-level 
productivity often rely on surveys of faculty to collect data on their publications, since 
this information is rarely collected centrally by IHEs. In contrast, data on sponsored 
research dollars received are readily accessible to analysts for virtually all institutions 
through the IPEDS surveys sponsored by the National Center for Education Statistics. 

Measuring an Institution’s Publications 

In order to help bridge the gap between the treatment of research in 
individual/departmental studies and institutional studies, we show how readily-accessible 
data on publications compiled by the Institute for Scientific Inquiry (ISI) can be used to 
derive a proxy measure of an institution’s research output. The Institute is responsible 

3 

for producing the Science Citation Index , the Social Science Citation Index, and the Arts 
and Humanities Index. While the three indexes do not include all academic journals in 
each field, they are fairly comprehensive in their coverage; together they include over 
6,600 scholarly journals in over 200 academic disciplines. Many institution libraries 
receive annual editions of the citation indexes in CD-ROM format, and other institutions 
subscribe to an on-line version of the databases. More details on the ISI databases and 
their coverage can be found on the ISI website (http://www.isinet.com/isi/) . 
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We used the data contained in the ISI databases to estimate the total number of 



publications that were attributed to individuals at each of approximately 1,300 four-year 
colleges and universities in the United States for 1996. Separate searches were conducted 
for the Science Citation Index, Social Science Citation Index, and Arts & Humanities 
Index. We restricted our searches to English-language articles published in academic 
journals during the calendar year 1996. To obtain publication counts for each institution, 
we began by identifying the number of journal articles where one or more authors listed 
the institution as their affiliation. From this set, we only counted those publications 
where either the name of the town or the zip code matched the town/zip codes that 
normally correspond with the institution. If publication counts were based solely on the 
institution name, then institutions with branch campuses or having the same name as an 
institution in another state would have inflated publication counts. The initial set of town 
names and zip codes for each institution were obtained from the Higher Education 
Directory (Rodenhouse, 2001), and then other towns and zip codes were added based on 
a manual review of the publications found for each institution name. We also reviewed 
all of the remaining publications to ensure that they either were associated with a branch 
campus of the institution in question or were from an institution with the same name but 
in another state. The search process was then repeated for other possible variations of 
institutional names. A visual depiction of this process is shown in Figure 1: 

y n Figure 1 Here 

The counting process is perhaps best illustrated with an example. In determining 
the number of publications for individuals at the University of Maine at Orono in 1996, 
an initial search revealed 404 journal articles where the affiliation of one or more authors 
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was the “University of Maine.” The University of Maine resides in the town of Orono, 
and has the zip codes 04469 and 04473 attached to it. When we searched on “University 
of Maine” and either “Orono” or “04469 or 04473,” we found a total of 228 journal 
articles. This was the final count that we used for this institution. 

There are, of course, some important limitations to this approach that must be 
explained before proceeding. First, our searches of the ISI databases only included 
scholarly journals, and thus exclude many other forms of publication including books, 
book reviews, corrections, editorial material, letters, etc. Thus institutions that have a 
comparatively large focus on arts or humanities may appear less productive, as these 
areas place less emphasis on publishing in scholarly journals. Second, despite the 
impressive breadth of coverage of the ISI databases, there are other academic journals 
and publication outlets that are not monitored by the Institute and thus are not included in 
their databases. Third, publications that list a particular institution but have an address 
that is not regularly associated with the institution will not be included in the institution’s 
total publication count. This would pertain to faculty who are on sabbatical and might 
report a different town and zip code as their location. Fourth, publications that do not list 
the institution will not be included in the publication count. Fifth, publications are not 
weighted by number of authors, so a publication with three authors at three different 
institutions is counted three separate times, and is weighted the same as a single-author 
piece. 

Despite these limitations, we believe that the resulting publication counts provide 
a reasonable proxy for the institutional total. While some of the issues described here 
could be addressed through searches on lists of individuals, presuming that one could 
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obtain lists of appropriate individuals for a large number of schools, this would be 
extremely time-consuming and difficult to conduct, especially on an annual basis. 

In an effort to produce a more standardized measure of institutional research 
output, we divided the total publication counts by the number of full-time faculty to 
calculate the ratio of publications to full-time faculty. The faculty headcounts were 
obtained from the IPEDS Faculty Salary survey for the 1996 academic year, and include 
only full-time faculty at the ranks of Full Professor, Associate Professor, Assistant 
Professor, and Instructor. It is important to note, however, that this ratio should not be 
confused with the average publications per faculty for several reasons. First, as described 
earlier, not all publications will be included in the institution’s total. More importantly, 
some of the publications attributed to the institution may have been written by individuals 
who are affiliated with the institution but are not full-time faculty. This would potentially 
include graduate students, adjunct faculty, part-time but tenure-eligible faculty, other 
faculty not reported by institutions to IPEDS, and professional/ administrative staff who 
might be engaged in research. Finally, using this procedure means that the credit for 
publications that are co-authored by faculty members at the same institution is only 
attributed to one individual, even though each would claim some credit for the 
publication. 

Empirical Results 

Figure 2 shows the distribution of institutional publications, along with selected 
descriptive statistics for the whole sample. Not surprisingly, the distribution of 
institutional publications is highly skewed to the right, with the skewness being even 
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greater than is apparent in the figure since institutions are placed into groups of unequal 
size based on their publication level. This skewness can also be seen by the fact that the 
mean number of publications (169) greatly exceeds the median (6). Close to 20% of the 
1,309 four- year institutions in our analysis had zero publications in 1996, and at the other 
extreme, only 13% of the institutions were found to have 200 or more publications. 

5k********** Jxiscri Figure ^ Here 

Figure 3 presents similar descriptive information on the ratio of publications to 
faculty for the institutions in our sample. Due to missing values for some institutions on 
the number of faculty, our sample decreases to 1,283 institutions. Again, we found that 
the distribution of the ratio of publications to faculty is highly skewed to the right, with 
two-thirds of the institutions having a ratio of 0.10 or lower (meaning that there are ten or 
more faculty for every one publication). Likewise, less than 7 percent of the institutions 
have a ratio of publications to faculty than exceeds one. 

***************** pig Ure ^ rc ***************** 

We next ranked all of the institutions from highest to lowest according to total 
publications in 1996, and list the top 100 institutions in descending order in Table 1. 
Harvard University was easily the leading producer of total publications, with 7,243 for 
the calendar year 1996. The University of California System placed four of its 
institutions among the top 20 in the nation, with UCLA placing second, UC-Berkeley 
ranking third, UC-San Diego 1 1 th , and UC-Davis in 17 th place. Other institutions that 
appeared in the top 10 based on their total publications were the University of 
Washington, the University of Minnesota, Stanford University, The University of 
Michigan, the University of Wisconsin, Johns Hopkins University, and the University of 
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Pennsylvania. The rankings based on total publications are similar to those produced by 
TheCenter, with 19 institutions appearing in the top 24 for both lists. 

******* ******** insert Table 1 Here ************** 

One limitation of the rankings of institutions shown in Table 1, however, is that 
they will be biased towards larger institutions since total publications are not expressed 
on a per-capita basis. Accordingly, we also ranked institutions from high to low on their 
ratio of publications to full-time faculty. These results are shown in Table 2: 

While many of the institutions that are highly ranked on the basis of total publications are 
also highly ranked according to the ratio of publications to full-time faculty, some smaller 
institutions that are nonetheless heavily involved in research rank higher in terms of per- 
capita research production. We found that California Institute of Technology had the 
highest ratio of publications to full-time faculty (7.63) and easily outpaced the second 
place institution Harvard (6.48), even though they were in 37 th place based on total 
publications. Likewise, the University of Texas at Dallas had the third-highest ratio of 
publications to full-time faculty, while they were ranked 58 th in terms of total 
publications. Other “smaller” institutions that managed to move up in the rankings when 
output was expressed on a per-capita basis include the University of Maryland at 
Baltimore County (12 th ), the University of Rochester (16 th ), Tufts University (18 th ), the 
University of Texas at San Antonio (20 th ), Wake Forest University (27 th ), UC-Riverside 
(34 th ), and Yeshiva University (41 st ). 

Rankings based on the ratio of publications to full-time faculty give rise to 
notably different rankings than those shown by TheCenter, with only 12 institutions 



appearing in the top 24 for both lists. In particular, larger institutions such as the 
University of Michigan, the University of Minnesota, and Columbia University tended to 
be lower when based on the ratio of publications to full-time faculty. In Table 3, we 
provide a comparison of how the top 10 institutions according to total publications or the 
ratio of publications to full-time faculty compare to the Top 10 lists from three alternative 
institutional rankings: (1) TheCenter’s research university rankings for 2001 (Lombardi 
et al., 2001), (2) the average scholarly quality of faculty in 1993 according to the National 
Research Council study (Webster and Skinner, 1996; Diamond and Graham, 2000), and 
(3) the overall institutional score in 1996 for national universities according to USNWR 
(1998). The first column shows the top 10 institutions when ranked according to either 
total publications or the ratio of publications to full-time faculty. The numbers shown in 
each column denote the respective ranking for the same institution in the alternative 
rankings. USNWR also groups institutions into four “tiers,” with tier 1 being those 
institutions with the highest ranking and tier 4 the lowest rankings. The numbers shown 
for most of the institutions in the USNWR column denote their ranking within the top 
tier. 

T'clbl.G 3 HOl*0 

One might expect some differences in rankings to emerge due to the timing at 
which the various measures were produced (1993, 1996, and 2001) and the types of 
metrics being examined. Nonetheless, there is a fair amount of overlap between the 
rankings based on research output and the other three rankings. The rankings produced 
by TheCenter clearly are more closely aligned with the institutional rankings based on 
total publications. Interestingly, the USNWR rankings are the most similar to the 
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rankings based on the ratio of publications to full-time faculty. Except for the University 
of Texas at Dallas, most of the institutions that were in the top 10 on the basis of total 
publications or the ratio of publications to full-time faculty rated highly in the other lists 
as well. 

Correlates of Research Productivity. When looking at the research output of a 
large number of institutions, it is important to acknowledge that not all IHEs place the 
same emphasis on teaching and research. This emphasis is likely to vary according to 
mission, which can be reflected in part by classification. In Table 4, we provide 
breakdowns of average total publications and average ratios of publications to faculty by 
type of institution. The breakdowns are shown for both the Carnegie classifications of 
institutions in 1996 as well as the classification scheme used by USNWR : 



Table 4 shows that, not surprisingly, the vast majority of research is produced by 
research- and doctoral-level institutions. Likewise, institutions that are not in the 
National University category for USNWR on average produce fewer than 30 publications 
per year. Interestingly, we can see that the Carnegie classifications are not a completely 
linear predictor of research productivity. The ratio of publications to faculty drop as we 
move down from Research I institutions to Doctoral I institutions, but Doctoral II 
institutions produce more publications per faculty member than Doctoral I institutions. 

In addition, Baccalaureate I institutions are slightly more productive than both Masters I 
and Masters 13 institutions, while Masters II and Baccalaureate II institutions appear very 
similar. 



***************** Jnscrt Table 4 Here ***************** 



ERIC 




19 



In Table 5 we explore how these two measures of research output are correlated 
with commonly used research measures and other popular higher education quality 
indicators. The first variable is the level of research expenditures at the institution, as 
reported on line 023 of the IPEDS Finance Survey, Part B. As we mentioned in the 
Introduction, this quantity understates the true level of research expenditures at the 
institution since some portion of faculty salaries, normally reported on line BO 13, are also 
used to support research. The second variable we examine is the level of federal research 
dollars received by the institution (line 063 from IPEDS Finance Survey, Part A). We 
created two additional variables by dividing these dollar totals by the number of full-time 
faculty, in an effort to express these metrics on a per-capita basis, and also considered the 
average salary for Full Professors. The final three variables we focus on here are not 
directly related to the research activities of an institution, but are often considered to be 
measures of institutional quality. These three factors, all obtained from the 1998 edition 
of USNWR, are the average six-year graduation rate for the institution, the percentage of 
applicants admitted to the institution, and the 25 th percentile of SAT scores for freshmen. 

The correlations in Table 5 show that the two institutional publication output 
measures we present here are highly correlated with the level of research expenditures 
and revenues at the institution. Significant, but smaller, correlations were found between 
the two publication measures and average faculty salaries. Finally, with the exception of 
freshmen SAT scores, we found little evidence that publications were correlated with the 
three non-research measures that we considered. Although statistically significant, the 
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correlations for SAT scores are quite low, especially in comparison to the correlations for 
research money and salaries. 

Uses for Institutional Research. Institutional researchers, especially those 
employed by doctoral-granting institutions, would also find this measure to be valuable in 
their efforts to develop meaningful, output-related strategic indicators for their institution. 
The University System of New Hampshire, for example, uses data on institutional 
publications to track changes in the ratio of publications to faculty over time, and to 
compare the ratio for our institutions to their officially adopted comparator institutions. 
This information is reported on an annual basis to the Board of Trustees and other 
education stakeholders through the USNH Strategic Indicator report. Table 6 shows, for 
example, how the University of New Hampshire compares in terms of their ratio of 
publications to faculty to the other five land-grant universities in New England. 

************* Jnscrt Table 6 Here ******************** 

Summary and Discussion 

In this study, we showed how available data on the publications of faculty can be 
used to derive institutional measures of research output. The construction of these 
measures has been aided in recent years by the release of data on CD-ROM and on-line 
databases, allowing users to quickly perform searches by institution. By combining 
publication data with information on the number of faculty at an institution, the total 
output measure can also be expressed on a per-capita basis. We have shown that this can 
be especially relevant for smaller institutions that still provide a significant amount of 
emphasis towards research. 
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It is worth reiterating at this point that the publication counts obtained through 
this procedure will likely understate the true quantity of publications attributed to a 
particular institution. Our methodology is meant to provide a reasonable proxy to this 
total that can be obtained in a consistent and fairly time-efficient manner. It is 
encouraging to note that the resulting publication measure is very highly correlated with 
other commonly-used measures of resources devoted to research, and that the rankings 
based on institutional publication counts do not produce results that are counterintuitive. 

Given the increasing emphasis on performance indicators, the measure of the ratio 
of publications to full-time faculty member can fill an important gap in how institutions 
are evaluated and compared. While measures of the teaching aspect of institutional 
mission are numerous (see, for example, the measures in the USNWR rankings), 
corresponding measures of the research aspect are few and far between. And while 
research grants and research expenditures are important components of most institutions, 
the research production of the faculty is the output that most closely corresponds with 
research aspect of institutional mission. Nonetheless, there is a very high correlation 
between the level of resources expended or received by institutions for research and the 
number of publications produced. 
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Figure 1: Venn Diagram of Search Process Used to Count Institutional 

Publications 



The goal is to isolate AB, 
ABC, AC, and then BD, 
BCD, and CD . 



A = Institutional Name 
B = City/s 
C = Zip/s 

D = Alternative Institutional 
Names/ Abbreviations 
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Figure 2: Frequency Distribution of Total Publications by Institution, 1996 




Mean = 168.73 
Median = 6.00 
Standard Deviation = 554.56 
Minimum = 0 
Maximum = 7,243 
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Figure 3: Frequency Distribution of Ratio of Publications to Faculty, 1996 




Ratio of Publications to Faculty 



Mean = 0.266 
Median = 0.056 
Standard Deviation = 0.669 
Minimum = 0 
Maximum = 7.63 
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Table 1: Total Publications by Institution, 1996 


Rank 


Institution 


# Pubs 


Rank 


Institution 


# Pubs 


1 


Harvard University 


7,243 


51 


U. Alabama-Birmingham 


1,446 


2 


UC-Los Angeles 


4,428 


52 


University of Georgia 


1,437 


3 


UC-Berkeley 


3,900 


53 


Iowa State University 


1,433 


4 


University of Washington 


3,797 


54 


U. Colorado-Boulder 


1,404 


5 


U. Minnesota-Twin Cities 


3,783 


55 


Northwestern University 


1,381 


6 


Stanford University 


3,759 


56 


UC-Santa Barbara 


1,372 


7 


University of Michigan 


3,758 


57 


Wayne State University 


1,324 


8 


Univ. of Wisconsin-Madison 


3,634 


58 


University of Texas-Dallas 


1,317 


9 


Johns Hopkins University 


3,420 


59 


Brown University 


1,310 


10 


University of Pennsylvania 


3,417 


60 


U. Maryland-Baltimore Cnty 


1,308 


11 


UC-San Diego 


3,264 


61 


Tufts University 


1,295 


12 


University of Pittsburgh 


3,110 


62 


Virginia Tech 


1,259 


13 


Columbia University 


2,982 


63 


Tulane University 


1,248 


14 


MIT 


2,920 


64 


Georgia Institute of Tech. 


1,175 


15 


Duke University 


2,911 


65 


Indiana University 


1,151 


16 


Yale University 


2,841 


66 


Arizona State University 


1,150 


17 


UC-Davis 


2,774 


67 


U. Missouri-Columbia 


1,147 


18 


University of Florida 


2,724 


68 


SUNY-Buffalo 


1,142 


19 


U. Illinois 


2,584 


69 


Louisiana State U. 


1,098 


20 


Emory University 


2,563 


70 


Colorado State University 


1,035 


21 


Cornell University 


2,447 


71 


Univ. of Tennessee-Knoxville 


1,006 


22 


University of Chicago 


2,389 


72 


University of New Mexico 


1,005 


23 


Ohio State University 


2,339 


73 


U. Texas-San Antonio 


1,001 


24 


Washington University 


2,337 


74 


Virginia Commonwealth U. 


967 


25 


University of Arizona 


2,266 


75 


Georgetown University 


961 


26 


Univ. of Southern California 


2,262 


76 


U. Massachusetts-Amherst 


938 


27 


Pennsylvania State U. 


2,185 


77 


Carnegie Mellon University 


933 


28 


Michigan State U. 


2,174 


78 


Oregon State University 


917 


29 


U of North Carolina 


2,166 


79 


University of South Florida 


903 


30 


University of Iowa 


2,151 


80 


University of Nebraska 


901 


31 


University of Texas-Austin 


2,133 


81 


Washington State University 


882 


32 


Texas A&M Univ. College 


2,117 


82 


Florida State University 


845 


33 


U. Maryland-College Park 


2,044 


83 


University of Connecticut 


840 


34 


Purdue Univ-West Lafayette 


2,006 


84 


Temple University 


837 


35 


Case Western Reserve U. 


1,934 


85 


University of Delaware 


801 


36 


New York University 


1,908 


86 


UC-Riverside 


784 


37 


California Institute of Tech. 


1,900 


87 


University of Houston 


761 


38 


University of Virginia 


1,837 


88 


Kansas State University 


754 


39 


Rutgers-New Brunswick 


1,740 


89 


Auburn University 


753 


40 


Boston University 


1,664 


90 


Wake Forest University 


752 


41 


U. Illinois-Chicago 


1,655 


91 


U. South Carolina-Columbia 


687 


42 


University of Utah 


1,633 


92 


George Washington U. 


682 


43 


University of Rochester 


1,584 


93 


University of Kansas 


666 


44 


UC-Irvine 


1,571 


94 


St. Louis Univ. 


640 


45 


SUNY-Stony Brook 


1,561 


95 


University of New Orleans 


636 


46 


North Carolina State U. 


1,519 


96 


West Virginia University 


625 


47 


Princeton University 


1,509 


97 


UC-Santa Cruz 


612 


48 


Vanderbilt University 


1,500 


98 


University of Vermont 


605 


49 


University of Cincinnati 


1,458 


99 


University of Nebraska 


601 


50 


University of Kentucky 


1,455 


100 


University of Notre Dame 


599 
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Table 2: Top 50 Institutions Based on Ratio of Publicati 


ions to Full-Time Faculty 


Rank 


Institution 


Number of 
Publications 


Number of full- 
time faculty 


Publications to 
FT Faculty 


1 


California Institute of Technology 


1,900 


249 


7.63 


2 


Harvard University 


7,243 


1117 


6.48 


3 


University of Texas-Dallas 


1,317 


207 


6.36 


4 


Johns Hopkins University 


3,420 


666 


5.14 


5 


Univ. of California-San Diego 


3,264 


644 


5.07 


6 


Emory University 


2,563 


546 


4.69 


7 


Stanford University 


3,759 


827 


4.55 


8 


Duke University 


2,911 


649 


4.49 


9 


Yale University 


2,841 


718 


3.96 


10 


Washington University 


2,337 


604 


3.87 


II 


Case Western Reserve Univ. 


1,934 


519 


3.73 


12 


U. of Maryland-Baltimore County 


1,308 


354 


3.69 


13 


University of Pennsylvania 


3,417 


977 


3.50 


14 


Univ. of California-Los Angeles 


4,428 


1311 


3.38 


15 


University of California-Berkeley 


3,900 


1161 


3.36 


16 


University of Rochester 


1,584 


489 


3.24 


17 


University of California-Davis 


2,774 


861 


3.22 


18 


Tufts University 


1,295 


413 


3.14 


19 


Massachusetts Inst, of Technology 


2,920 


936 


3.12 


20 


University of Texas-San Antonio 


1,001 


326 


3.07 


21 


University of Chicago 


2,389 


814 


2.93 


22 


Tulane University 


1,248 


436 


2.86 


23 


University of California-Irvine 


1,571 


572 


2.75 


24 


Brown University 


1,310 


488 


2.68 


25 


Univ. of Minnesota-Twin Cities 


3,783 


1499 


2.52 


26 


University of Pittsburgh 


3,110 


1252 


2.48 


27 


Wake Forest University 


752 


318 


2.36 


28 


Univ. of Wisconsin-Madison 


3,634 


1546 


2.35 


29 


Univ. of California-Santa Barbara 


1,372 


593 


2.31 


30 


Vanderbilt University 


1,500 


653 


2.30 


31 


Princeton University 


1,509 


662 


2.28 


32 


University of Washington 


3,797 


1698 


2.24 


33 


University of Michigan-Ann Arbor 


3,758 


1712 


2.20 


34 


Univ. of California-Riverside 


784 


371 


2.11 


35 


University of Alabama-Birmingham 


1,446 


714 


2.03 


36 


SUNY-Stony Brook 


1,561 


780 


2.00 


37 


University of Iowa 


2,151 


1080 


1.99 


38 


Univ. of Southern California 


2,262 


1165 


1.94 


39 


University of Virginia 


1,837 


955 


1.92 


40 


U of North Carolina-Chapel Hill 


2,166 


1156 


1.87 


41 


Yeshiva University 


387 


208 


1.86 


42 


Georgia Institute of Technology 


1,175 


634 


1.85 


43 


Carnegie Mellon University 


933 


507 


1.84 


44 


University of Utah 


1,633 


905 


1.80 


45 


Georgetown University 


961 


546 


1.76 


46 


University of Arizona 


2,266 


1303 


1.74 


47 


Cornell University 


2,447 


1417 


1.73 


48 


Wayne State University 


1,324 


784 


1.69 


49 


Univ. of Maryland-College Park 


2,044 


1221 


1.67 


50 


Univ. of California-Santa Cruz 


612 


371 


1.65 
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Table 3: Comparison of Alternative Institutional Rankings 



< Rankings Based on > 

Total Publications 
Rank Institution 

1 Harvard University 

2 UC-Los Angeles 

3 UC-Berkeley 

4 U Washington 

5 U Minnesota 

6 Stanford University 

7 U Michigan 

8 U Wisconsin 

9 Johns Hopkins U 

10 U Pennsylvania 

< — Rankings Based on Ratio — > 
Publications to FT Faculty 
Rank Institution 

1 California Institute of Tech. 

2 Harvard University 

3 U Texas - Dallas 

4 Johns Hopkins U 

5 UC - San Diego 

6 Emory University 

7 Stanford University 

8 Duke University 

9 Yale University 

10 Washington U 



< 


- Rankings Based on — 


> 




NRC Reputation 


USNWR 


TheCenter Til Score T21 


Overall T31 


i 


3 


1 


12 


12 


28 


8 


2 


23 


15 


17 


Tier 2 


8 


23 


Tier 2 


1 


6 


5 


8 


12 


23 


12 


15 


38 


6 


20 


14 


1 


14 


7 



<* nanmngs nasea on — 

NRC Reputation 


> 

USNWR 


TheCenter I’ll 


Score T21 


Overall T31 


26 


4 


9 


1 


3 


1 


50+ 


50+ 


Tier 3 


6 


20 


14 


21 


10 


33 


38 


32 


9 


1 


6 


5 


8 


20 


3 


16 


8 


3 


16 


34 


17 



Notes: [1] Rankings of research universities for 2001 as derived by TheCenter (Lombardi et al., 2001). 

The Top American Research Universities . Gainesville, FL: TheCenter, University of Florida. 

[2] Rankings for 1993 based on the average scholarly quality of faculty, 1993 NRC graduate program 
survey. See Diamond and Graham (2001), Table 11. 

[3] Rankings based on overall score in 1996 as reported by U.S. News and World Report (1998). 
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Table 4: 



Breakdown of Mean Total Publications and Ratio of Publications to 
Full-Time Faculty by Institution Type 



Institution Tvoe 


Mean: Total 
Publications 


Mean: Ratio of 
Publications 
to Faculty 


Number of 
Institutions 


Carnegie Classification , 


Research I 


1,886 


2.04 


83 . 


Research II 


548 


0.91 


36 


Doctoral I 


195 


0.46 


48 


Doctoral II 


205 


0.56 


54 


Masters I 


32 


0.11 


393 


Masters II 


4 


0.04 


87 


Baccalaureate I 


19 


0.14 


156 


Baccalaureate II 


2 


0.03 


395 


USNWR Classifications 


National Universities 


896 


1.15 


227. 


National Liberal Arts 


18 


0.14 


147 


Regional Universities 


27 


0.10 


500 


Regional Liberal Arts 


2 


0.03 


421 


All Institutions 


169 


0.27 


1,309 
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Table 5: 



Correlations of Total Publications and Ratio of Publications to Full- 
Time Faculty with Other Measures 



Variable 



Ratio: 

Total Publications 

Publications to FT Faculty 



Research Expenditures 
Research Expenditures/Faculty 
Federal Research Revenues 
Federal Research Revenues/Faculty 
Average Full Professor Salary 
Six-Year Graduation Rate 
Freshman Acceptance Rate 
Freshman SAT Score (25 th ) 



+0.92** 


+0.74** 


+0.77** 


+0.81** 


+0.91** 


+0.77** 


+0.63** 


+0.72** 


+0.41** 


+0.47** 


+0.03 


+0.04 


+0.03 


+0.03 



+0.15** +0.17** 

Notes: ** p < .01 (two-tailed test). Data on research expenditures, federal research revenues, number of 
faculty, and average full professor salary are taken from IPEDS Finance and Faculty Salary surveys for 
FY96. Data on the six-year graduation rate, freshman acceptance rate, and freshman SAT score are taken 
from U.S. News and World Report (1998). 
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Table 6: UNH and New England Land Grant Institutions 

Ratio of Publications to Faculty for Calendar Year 2000 



Total Publications 

Social Arts & Number Ratio: 

Science Science Humanities of Publications 



Institution [1] 

University of Vermont 525 

Univ of Massachusetts - Amherst 890 

University of Connecticut - Storrs 675 

University of Maine - Orono 228 

University of New Hampshire 306 

University of Rhode Island 204 



Mean (w/o UNH) 

UNH relative to Mean 



[2] 


[3] 


Total 


Faculty 


per Faculty 


98 


16 


639 


440 


1.45 


174 


34 


1098 


1049 


1.05 


202 


33 


910 


994 


0.92 


43 


5 


276 


327 


0.84 


73 


7 


386 


546 


0.71 


61 


6 


271 


623 


0.43 



0.94 

-0.23 



UNH Rank (1 = high) 5 

Source for both tables; 

Institute for Scientific Information (ISI) Citation Databases, 2000 - based on calendar year 
Number of faculty data from Academe, Mar/Apr 200 1 
Note: Number of Faculty includes Full Time only for both 9/10 month and 11/12 month contracts. 

[1] Science Citation Index Expanded indexes more than 5,800 major journals across 164 scientific disciplines. 

[2] Social Sciences Citation Index includes more than 1,725 journals spanning 50 disciplines. 

[3] Arts and Humanities Citation Index is a multidisciplinary database that indexes 1,144 journals. 

Data sorted on Ratio: Publications per Faculty 
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